### Building a Wiki in Haskell
#### A demonstration of functional web programming.

@author Sebastiaan Visser 

@email  sfvisser@cs.uu.nl

**! Possibly due to the immense popularity of Wikipedia[#ref-wiki| (#)], sharing
information online using a wiki has become more and more common.  Wikis are
document centric content management systems allowing multiple users to
contribute to the same collection of documents.  Generally, wikis expose
documents to the outside world using the world wide web.  This document
describes a wiki system written in the pure and functional programming language
Haskell. 

@@toc

*** Introduction

The intention of this document is to give an overall introduction to this wiki
project. It describes the functional aspects and gives a technical overview of
the software architecture.  The section on the system architecture contains
some references into the source code, intended to encourage people to look at
the source and maybe even start hacking.  Because the project is still in
development this document also describes the current status and gives an
overview of what still has to be done.

Briefly explained, this project consists out of three separate parts.  At the
core there is a generic application server allowing web applications to expose
there functionality over the internet. This application implements the HTTP
protocol.  The application server is written entirely in Haskell.  On top this
core is the actual wiki application.  This application enables the manipulation
of a specialized online document repository.  A graphical user interface on top
of this application gives users control over the document repository using a
regular web page.

This project aims to be a demonstration of Haskell, and functional programming
in general, as a powerful tool when it comes to web programming.  Can a pure
and lazy language like Haskell manage tho keep its head up when used in a world
full of impurity and side effects, the world wide web?

*** Properties

There are currently a lot of wiki applications available.  By looking at these
existing systems, we can easily extract a set of properties we expect a wiki to
have. 

- *CRUD.* This term stand for _create_, _read_, _update_ and _delete_.  We
  would like to manage documents with more or less the same freedom as we have
  on a regular file system in our operating system.  Though, this functionality
  should be available using a web interface.

- *Formatting.* We want to be able to structure our documents and add some
  minor markup.  We want to be able to divide our documents into sections,
  subsection, paragraphs, lists, etc.  We need a, preferably easy to use,
  formatting language that is powerful enough to support both structuring and
  formatting of documents.  Because even the untrained eye can read it, a
  format as close to plain text is most suitable.

- *Hyper-linking.* We should be able to include hyper-links to other documents,
  in order to navigate through a document collection.  To add more structure to
  a document collection it should be able to group them into a hierarchical
  directory-like structure using references.

- *Content.* In order to give the documents more flair we should be able to
  insert media into documents like images, tables, lists, source code snippets,
  mathematical formulas and possibly even more.

- *Version control.* In order to keep track of the modification history of
  documents we require a simple form of version control. This will help keeping
  track of changes and simplify cooperation and conflict resolution.

- *Output formats.* Documents should be viewable online in a formatted way, not
  the original format is was edited in.  By default the pages should be
  outputted in a nicely designed (X)HTML page. Additionally, there are lots of
  applications we can think of that would gain benefit from other output
  formats.

This project implements most of these properties and supplies a modular
framework in which new features can be build.  This application tries to be as
configurable and extensible as possible, allowing to extend the project without
touching the original code.  The ability of Haskell to create powerful
abstractions plays a crucial role in this modularity.  The next section on the
architecture explains how this is all achieved technically.

By default all documents are made publicly available online using a simple web
interface.  At first sight this interface looks more or less like a file system
browser as included in most web server environments.  This is because the wiki
document repository actually is represented as a hierarchical directory
structure.  This structure is though, more restricted at some points and
extended at other points.

The wiki server application implements a series of special request hooks that
influence the way documents are exposed to the web. These hooks enable
manipulating the documents and querying for additional information.

  *** Web service

  The wiki web interface can be seen as a so called Representational State
  Transfer (REST) interface.  REST interfaces are characterized by the ability
  to both retrieve documents from the server and to store new or manipulated
  document to the server.  These interface are not specifically intended to be
  used by end-users, but are more like web-services. 

  Differentiation of action is based on the HTTP request method.  A standard
  HTTP GET request is identified as retrieval while the PUT request can be used
  to store documents. The DELETE request method is used for removal.  REST
  interfaces are standard way to implement the CRUD pattern in a web service.
  This is very appropriate for the wiki document repository.

  The same dispatching mechanism can be used to map the requested document's
  file extension to an appropriate output format.  The file name part of the
  request URI is queried for the file extension, this extension is used to map
  to the appropriate wiki output handler.

  Currently there are six different output formats supported:

  - *.txt* This default extension shows the document in the formatting language
    it was originally edited in. No parsing or pretty printing is needed for
    this view.  It is currently only possible to create documents in the wiki
    format language, so no ambiguity is possible when reading the source
    document.

  - *.html* This extension forces the retrieval of a proper (X)HTML document.
    This is the default view on documents in the collection.  The original
    document in the plain text format is parsed and pretty printed as (X)HTML.
    All hyper-links and image paths are translated in order to be correctly
    navigateable online.

  - *.tex* This extension forces the document to be pretty printed as a LaTeX
    document.  This can be useful when creating documents that are not
    especially meant to be viewed online.  The LaTeX pretty printer is not
    entirely finished at the moment, but implements a usable set of constructs.
    Currently it is not possible to supply custom LaTeX templates in with the
    generated document should be included.

  - *.pdf* This forces the document to be pretty printed to LaTeX and
    successively converted to a PDF document.  This is a post processing step
    to the LaTeX pretty printer.

  - *.hs* Using this extension the document is printed using the regular
    Haskell =show= function.  It will show a direct dump of the internal
    document data structure.

  - *.his* This extension is somewhat special because it is not simply a
    different view on the document structure. It is used to query the version
    control back-end for the modification history of a document.  This will
    show a list of historic changes to the document.  This is a simple line
    separated plain text format that includes the change name, modification
    date and author. This list is by default ordered by modification date.

  Another dispatching mechanism is used that maps the URI query string (the
  part after the question mark) to a modification name in the document history.
  The query string is matched against the revision names in the version control
  back-end, when it matches a historic version is shown.  The query string can
  be used in combination with a custom file extension to query historic version
  in alternative formats.

  As explained above, the PUT request method is used for storing new documents
  in the collection. Because browsers do not allow end-users to store documents
  at the server using a HTTP PUT method directly, a special front-end
  application must be used.  When a new document is saved it will automatically
  be stored in version control.  Modifications to existing documents will also
  be stored in the version control system.  The query string for the request
  URI is used to determine the name of the change.  All save actions will
  immediately be reflected in the document history.

  The version control system that the client can observe is an abstract system.
  It exposes a normalized view on document history which does not necessarily
  reflect the concrete back-end used at the server.  More on the technical
  impact of this can be found in the section on architecture.

  Three additional request handlers are bound to the wiki repository. One for
  logging in to the system, one for logging out and one for querying the login
  status.  By posting the proper username and password variables to the
  =/login= handler one can authenticate itself by the server.  Generally it is
  only possible to alter documents when logged in, although the implementers of
  the wiki are free to change this policy. Logging out can be done using the
  =/logout= handler and the current login and session information is made
  available using the =/loginfo= handler.

  *** Front-End

  To provide a more user friendly and more visually appealing user interface a
  web front-end has been designed. This front-end is view on the REST
  interface.  This user interface makes all the functionality supplied by the
  wiki application available in a single front end application.

  This web page is written in a combination of (X)HTML and JavaScript.  
  This so called Ajax application provides a single page interface, meaning
  that no page refreshes are needed while working.  It uses HTTP requests to
  internally query the wiki's REST interface.  This user interface has no other
  connection to the server rather than the REST interface itself.  

  The wiki back-end has no knowledge of the eventual presentation of the user
  interface.  This modular approach allows for the independent creation of more
  user interface front ends for the wiki application.  More on this can be
  found in the section on architecture.

*** Architecture

This section briefly explains the internal architecture of the project.  It
aims to give some insight into the inner working of both the application server
and the wiki application.

  *** Application Server

  An application server is a web server framework that enables arbitrary
  applications to share their content using the HTTP protocol.  Unlike most
  standard web servers an application server focusses more on being a platform
  for web application rather than serving static content.

  People familiar with Haskell may know Happs [#ref-happs| (#)], a full-fledged
  Haskell application server (hence the name) currently used in some real world
  applications.  The application described here can be seen as a light weight
  variant of Happs.

  The application server has two main responsibilities.  First of all it is
  responsible for handling all low level network traffic and interpreting the
  HTTP protocol.  Secondly it supplies higher level abstractions to deal with
  the low level communication.

  The low level protocol handling is the less interesting part.  It takes care
  of: parsing and interpreting HTTP messages and request URIs, interpreting
  additional header field containing information about content encodings and
  types, cookie information, etc.  This is the part where most web server are
  actually pretty much the same, they just follow the protocol specification.

  The more interesting part of the application server is the way client
  requests are translated into server responses.  The server provides a powerful
  combinator library that allows users to setup their own request handlers.
  Handlers are special functions or combinators that are used to configure the
  way requests are interpreted and how the server responds to them.  Request
  handlers actually configure the behaviour of your website.
  
  The handler mechanism is a very modular system that allows for lots of reuse
  of existing components and is easily extended with new handlers or
  combinators.  This approach is very much in the spirit of the Haskell language
  and is probably what makes the difference between this application server and
  most server written in imperative languages.

  Currently there are already quite a lot default handlers available.
  Abstractions for custom parsing and printing of HTTP messages, for the
  creation of default error responses, virtual hosting, URI rewriting, message
  redirection, sessions, cookies, authentication, document storage, URI and
  method dispatching, serving files and directories and logging. 

  Additionally, the server has some simple utility functions that simply the
  input and output on the network socket.  These functions help dealing with
  encoding and hide most of the inner workings of the server core.

    *** Server Configuration and Startup

    After starting up the server core goes into an infinite loop with the only
    goal to accept incoming client connection and to spawn a request handler
    for this.  This is very common pattern when it comes to server
    applications.

    The server requires a configuration to be able to startup.  It is a simple
    structure containing some static server configuration.

    @@ haskell

      data HttpdConfig =
         HttpdConfig {
           hostname   :: String        -- Server hostname.
         , email      :: String        -- Server admin email address.
         , listen     :: (HostAddress  -- Addres to bind to.
                         ,PortNumber)  -- Port to listen on.
         , backlog    :: Int           -- TCP backlog.
         , bufferSize :: Int           -- Serve chunck with size.
         }

    Starting the server also requires a request handler, determining how to
    respond to the client request.

    @@ haskell

      start :: HttpdConfig -> Handler () -> IO ()

    There is already a default configuration available with some useful default
    settings.  This is how you start a server with the handler `myHandler' at
    localhost, port 8080.

    @@ haskell
  
      main :: IO ()
      main = do
        addr <- inet_addr "127.0.0.1"
        Httpd.start
          Httpd.defaultConfig {
            Httpd.hostname = "localhost"
          , Httpd.email    = "sfvisser@cs.uu.nl"
          , Httpd.listen   = (addr, 8080)
          }
          myHandler

    *** Handlers and Request Context

    Request handlers are used to analyse the client request and to prepare a
    server response.  In order to access both the request and the response we
    introduce a server context.  Such a context is unique for every request and
    contains all information needed for handlers to perform their tasks.
    Request handlers are state transformed IO monads with a server context as
    state.

    @@ haskell

      type Handler a = StateT Context IO a

    A server context contains enough information for handlers to prepare
    sensible responses.  This includes some static server configuration, the
    client request and the network socket.  Additionally, the context contains
    a default response message that can be altered and a queue of potential
    send actions.

      @@ haskell

        data Context =
          Context {
            -- For reading.
            config   :: HttpdConfig
            , request  :: Message
            , sock     :: Handle
            , address  :: SockAddr
            -- For writing.
            , response :: Message
            , queue    :: [Handle -> IO ()]
            }

    By analysing the request message handlers can setup appropriate a response
    message and send response bodies by enqueuing one or more send actions.
    The server makes sure all enqueued send actions are applied to the client
    socket after sending the response header.

    As a nice example of a request handler we show the =hHead= handler.  Is
    demonstrates how a handler can inspect the context information in the state
    monad and alter the response.
    
    @@ haskell

      hHead :: Handler a -> Handler a
      hHead handler = do
        m <- gets (method . request)
        case m of
          HEAD -> withRequest (setMethod GET) handler
               <* putQueue []
          _    -> handler

    This is a handler that transforms another handler.  This function analyses
    the method part of a request message and checks whether this is the HTTP
    HEAD request. It uses the =gets= from the state monad function to read
    values from the state monad.  When the request is a HEAD request it
    executes the input handler as if the request was a regular GET request and
    empties the send queue afterwards.  The return value of the input handler
    computation is returned.  When the method is not a HEAD, this function acts
    like the identity function.  Note how you can freely mix monadic and
    applicative syntax for the handlers, both instances are supplied.

    *** Start Hacking?

    Some of you might be interested in playing around with the server code and
    start hacking for you own.  Because there is still very much work to do and
    this can be a nice introduction to Haskell web programming we encourage you
    to look at the source.

    In the =src/Protocol= directory you can find all the low level protocol code.
    Currently it contains code for the HTTP protocol, a Mime library, a URI
    parser and a implementation of the Cookie specification.  There is also a
    set of QuickCheck [#ref-quickcheck| (#)] test cases available that test the
    URI parser and pretty printer.  This is still far from full coverage but
    they form a nice start.

    The application server core can be found in the =src/Httpd= directory.  This
    code actually has a very simple setup.  At first, it contains some protocol
    independent network abstractions used by the application server.  Secondly
    there is the server implementation itself defining the configuration,
    context and handler data types and the code for the accept loop.  A third
    module contains a set of helper functions that simplifies the socket input
    and output.

    All the default request handlers can be found in =src/Httpd/Handlers=
    directory.  All handlers have there own module and, due to their monadic
    behaviour, are generally very easy to understand.

  *** Format Library

  A significant part of the wiki application is formed by the format library.
  This library is responsible for parsing, pretty printing and conversion of
  documents in a variety of formats.  Because this library can be used
  independently of the wiki application it is discussed here separately.

  The library contains, amongst other things, code to deal with XML, (X)HTML,
  JSON and LaTeX.  These libraries implement functionality to parse these
  formats into a data structure, work on that data structure and to pretty
  print this back to this format.

  Note that the XML library, and the (X)HTML library which is based on this, is
  currently very primitive and does only support pretty printing.  No parsing
  yet.  We are planning to replace this library entirely with the mature and
  powerful HaXML [#ref-haxml| (#)] library, freely available on Hackage
  [#ref-hackage| (#)].

    *** Document

    At the core of the format library is the =Document= data type.  This
    document is a hierarchically structured document format powerful enough to
    express the most fundamental structure and markup.  The basic structure is
    loosely based on existing document types like (X)HTML, LaTeX and DocBook
    [#ref-docbook| (#)] It is a modular document structure in the sense that
    people can write plug-ins to implement more functionality without touching
    the original (most of) code.

    To work with this document type we need parsers to, and pretty printers
    from this type for formats that users or front-end applications can deal
    with.  Currently there is a parser available for a simple wiki language and
    a pretty printer to (X)HTML and LaTeX.  The system is open for additional
    parsers and pretty printers for other formats, see the section on hacking
    for more information on this.

    *** Plug-ins

    Orthogonally to the modularity of separate parsers and printers stands the
    concept of extending the document with new semantic concepts.  These
    plug-ins add new functionality to the document and may implement there own
    parsers and printers for this aspect only. 
    
    The maintainer of a plug-in is responsible for implementing the parsers and
    printers for different formats.  This means that is not enforced that a
    plug-in parser or a pretty printer of _every_ existing format.  Partial
    implementations are allowed.

    Technically a plug-in can be seen as a possible parser for a specific
    format that possibly delivers a pretty printer for a specific format. In
    Haskell you can write this down as:

    @@ haskell

      type Parser a      = Format -> Maybe (Parser a)
      data PrettyPrinter = Format -> Maybe Output
                                                                                                          
      data Plugin = Parser PrettyPrinter

    Currently there are three plug-ins available:
    
    - *HsColour.* The first is the HsColour plug-in allowing Haskell code
      snippets to be included in the document.  The plug-in pretty prints the
      code snippets with syntax highlighting to (X)HTML (and possibly to LaTeX)
      using the HsColour [#ref-hscolour| (#)] library. 

    - *Formula.* This plug-in allows the inclusion of LaTeX snippets into the
      document.  When pretty printing to LaTeX these snippets are, trivially,
      directly included into the document.  When pretty printed to (X)HTML to
      LaTeX formulas are converted to PNG images which are included in the
      document.

    - *TOC*.  This plug-in allows the inclusion of one (or more) placeholders
      into your document at places you want a automatically generated `table of
      contents' to appear.  This works (out of the box) for LaTeX documents and
      has a custom implementation for the (X)HTML pretty printer.

    *** Start Hacking?

    It would always be nice to have more input and output formats available for
    the document type, or have more coverage for the existing parsers and
    pretty printers.  Because the format library is far from complete (and will
    probably never will) we encourage people to start hacking there own
    plug-ins for the system.

    You can start by looking at the =src/Format= directory.  The =src/Xml=,
    =src/Xhtml= and =src/Json= directories contain simple abstractions for
    dealing with these formats out of the context of the real document data
    type.

    The document data type can be found in the =src/Format/Document= directory.
    We use attribute grammars because they allow us to easily perform data type
    traversals in an aspect oriented way.  Because not is not yet trivially
    easy to parametrize attribute grammars it is not always easy to customize
    the way the plug-ins behave. 

    The example plug-ins in =src/Format/Document/Plugin= directory form a good
    start point when trying to implement a plug-in your own.  It is even
    possible to reuse the parsers from the base wiki parser module.

  *** Wiki Application

  The wiki application is built on top of the application server and implements
  a single request handler used as an entry point to the entire application.
  This handler performs request routing based on the URI file extension and the
  request method and uses this to implement the REST interface discussed in the
  previous sections.

  It might be interesting to see whether the abstractions used to implement the
  REST like interface can be generalized and reused in other applications.  But
  currently it is specifically implemented for the wiki application.

  The main wiki handler, called =hWiki=, has the following type signture:

  @@ haskell

    hWiki :: UserDatabase -> UserSession -> Backend -> Handler ()

  The first two arguments required by the wiki handler are a user database
  desribing what users are allowed to perform what actions in the application
  and a session associated with the current user.  The application uses the
  built-in session and authentication modules supplied by the application
  server.  You get authentication and authorization for free!

  The third argument the application requires is concrete back-end
  implementation.  This is a description of how the application stores, deletes
  and retrieves physical documents.  This back-end is discussed in more detail
  below.

    *** Output formats.

    The wiki application has a modular output format system.  This means that
    that adding new output formats to the application can be done without
    altering the wiki core application.  This system is very closely linked to
    the format library. 

    @@ haskell

      data Output =
          TextOutput   String
        | BinaryOutput B.ByteString

      data WikiFormat a = 
        WikiFormat {
          extension :: String
        , mime      :: String
        , handler   :: Backend -> U.URI -> String -> IO Output
        }

    For every format described earlier the application contains one
    =WikiFormat= instance.  So one instance is available for the source format,
    (X)HTML, LaTeX, Haskell data type dump, PDF and the special history format.

    *** Storage Back-end

    Not only the format system is modular, also the storage back-end is
    abstract.  The wiki application makes no assumptions about how the documents
    are actually stored, an abstract =Backend= data type is used to model the
    interface to a physical back-end.

    @@ haskell

      data Revision =
        Revision {
          date   :: Date
        , author :: Author
        , name   :: String
        }

      type History = [Revision]

      data Backend =
        Backend {
          store    :: FilePath -> Revision -> String -> IO ()
        , delete   :: FilePath -> Revision -> IO ()
        , retrieve :: FilePath -> Revision -> IO (Maybe String)
        , history  :: FilePath -> IO (Maybe History)
        }

    When implementing an back-end instance four function must be supplied.
    It must be able to store, delete and retrieve document with the given =FilePath=
    and =Revision= as an abstract identifier.  The back-end must be able to
    manage document history and supply this as a =History= type.

    Currently there is one physical back-end implementation and one simple
    combinator: 

      - *Darcs back-end.* By using a connection to the command line tools this
        back-end uses the Darcs[#ref-darcs| (#)] version control system to
        manage storage and versioning.
        
        @@ haskell

          darcsBackend :: FilePath -> FilePath -> Backend

        The back-end constructor takes a directory currently under Darcs
        version control and a directory to store temporary documents.  Because
        Darcs is written in Haskell as well it would be nice to have a pure
        library version of this back-end as well.

      - *Caching back-end.* This combinator is a function that takes a
        =Backend= to a new =Backend= that caches historic documents in a
        special cache directory on the file system.  This can speed up
        reoccurring queries to historic versions of a document.  The first
        argument indicates the cache directory.

        @@ haskell

          cachingBackend :: FilePath -> Backend -> Backend

    *** Start Hacking?

    By looking at the =src/Wiki= directory it becomes clear that the setup is
    very simple.  The =src/Backend= and the =src/Format= directories contain the
    plug-ins that are used by the wiki application.  Adding more back-ends, like
    GIT, Subversion or plain file system storage, should be possible without
    touching the wiki core.  Extending the set of formats probably involves
    extending the =src/Format= as well.

  *** Front End

  The front-end user interface is a simple in-browser JavaScript application
  written using the popular jQuery library.  Because this user interface only
  uses the REST interface as a back-end and the server has no knowledge of the
  front-end, adding new user interfaces will not be problem.

*** Future Work

As became clear in previous sections the system is far from finished; lots of
things can be improved or extended.  This section gives a brief, but somewhat
arbitrary, enumeration of work that can be done to improve the system.  This is
list is far from complete but gives some pointer into future work.

- *Testing.* To prevent us from introducing regression bugs into the system it
  would be nice have test case for important parts of the code.  Currently only
  very selective part of the system are tested.

- *Document.* Not all operation, like deletion, on the document are supported
  in an entirely clear way.  It would also be nice to have an include mechanism
  that allows the inclusion of external sections into a document

- *Hot plugging.* The ability to load new plug-ins into the program at runtime
  can be a useful addition.  Scripting the document system this way might lower
  the threshold for people to play with the system.

- *Parsers.* It would be nice to have more parsers that target the document
  data type.  Having an XML parser and pretty printer around might improve
  interoperability with other applications.  The wiki parser itself is not
  entirely correct at the moment, it still contains some glitches and
  unsupported corner cases.

- *Code refactor.* For maintenance reasons it is important to keep the code
  clean and have simple module interfaces.  Maybe it is possible to put the
  =Handler= monad also in the =MonadPlus= class, enabling easy setup of
  fall-back handlers in case of failure.  Lots of code can be generalized to
  more abstract definitions, this will improve reusability.

- *Versioning.* It is currently not defined what happens when multiple users
  edit the same document concurrently.  A better and safer integration with the
  version back-end may solve this issue.  Creating an interface for conflict
  resolution would really be helpful.  Additional back-ends for other version
  control systems, like GIT, Subversion or Mercurial, will be nice to have as
  well.

- *Search.* Essential but not currently implemented, a search engine that
  indexes the document collection.

- *User interface.* The user interface currently has a built-in list of
  capabilities of the server.  What formats are supported and what operations
  can be performed on what data.  It would somehow be nice if the server can
  explicitly export a list of capabilities the front-end can adapt its
  interface automatically.

- *Documentation.* The documentation should be improved. The source code must
  be decorated with more comments to be included into the generated Haddock
  [#ref-haddock| (#)] documentation.

*** Conclusion

While the system is not yet finished it does currently work.  By abstracting
away implementation details in generic data types it is possible to create a
very modular and extensible system. Using small combinator functions to
construct domain specific languages for core functionality increases the
simplicity of the system without losing configurability.

The only reasonable conclusion to draw here is that Haskell is very well suited
for web programming.

*** Appendix A - List of Packages

+ #anchor ref-darcs

  Darcs. http://www.darcs.net/

+ #anchor ref-docbook

  DocBook. http://www.docbook.org/

+ #anchor ref-haxml

  HaXML. http://www.cs.york.ac.uk/fp/HaXml/

+ #anchor ref-hackage

  Hackage. http://hackage.haskell.org/packages/hackage.html

+ #anchor ref-haddock

  Haddock. http://www.haskell.org/haddock/

+ #anchor ref-happs

  Happs. http://happs.org/

+ #anchor ref-hscolour

  HsColour. http://www.cs.york.ac.uk/fp/darcs/hscolour/

+ #anchor ref-quickcheck

  QuickCheck. http://www.cs.chalmers.se/~rjmh/QuickCheck/

+ #anchor ref-wiki

  Wikipedia. http://www.wikipedia.org/

